Clock syntonization using network effect and/or adaptive stochastic control

ABSTRACT

Systems and methods are disclosed herein for syntonizing machines in a network. A coordinator accesses probe records for probes transmitted at different times between pairs of machines in the mesh network. For different pairs of machines, the coordinator estimates the drift between the pair of machines based on the transit times of probes transmitted between the pair of machines as indicated by the probe records. For different loops of at least three machines in the mesh network, the coordinator calculates a loop drift error based on a sum of the estimated drifts between pairs of machines around the loop and adjusts the estimated absolute drifts of the machines based on the loop drift errors. Here, the absolute drift is defined relative to a drift of a reference machine.

TECHNICAL FIELD

This disclosure relates generally to clock syntonization (i.e.,correcting frequency drift) of a local clock of a machine relative to areference clock, for example using network observations and/or adaptivestochastic control.

DESCRIPTION OF THE RELATED ART

Limits to clock synchronization accuracy between machines (e.g., in adata center) impose practical limitations in many applications. Forexample, in finance and e-commerce, clock synchronization is crucial fordetermining transaction order, in that a trading platform must matchbids and offers in the order in which those bids and offers are placed.If clocks of machines used to submit or route the bids and offers arenot synchronized, then bids and offers may be matched out of order,which results in a lack of fairness. Similar problems occur in othernetworked computer systems, such as distributed databases, distributedledgers (e.g., blockchain), distributed transaction tracing systems,distributed snapshotting of computation or networks, 5G mobile networkcommunications, and so on. In these systems, limits on clocksynchronization result in jitter, which results in biased or non-optimalprocessing of communications.

Related art attempts to achieve accuracy in clock synchronization on theorder of one hundred nanoseconds or better are generally expensive andimpractical, as these attempts require specially designed hardware to beimplemented throughout a network for combatting random network delays,component noise, and the like.

Within clock synchronization, clock syntonization (i.e., matching thefrequency of different clocks) is an important subtask. Even if twoclocks are perfectly synchronized at an instant in time, they will driftout of synchronization if their clock frequencies are not also matched(i.e., syntonized). Because clock frequencies can drift due to thermaland other environmental factors, clock syntonization is also generallyexpensive and impractical.

As a result, nanosecond-level clock synchronization is rarely used and,instead, algorithms that achieve millisecond-level accuracy withoutrequiring specialized equipment throughout the network have become thenorm (e.g., Network Time Protocol (NTP)). The world has simply come toaccept and tolerate the lack of fairness inherent in the technicallimitations of those millisecond-level synchronization solutions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a graph of a network including a coordinator for executing anembodiment of the disclosure.

FIG. 2 is a data flow diagram for correcting clock frequency and/oroffset, according to an embodiment of the disclosure.

FIG. 3 is a block diagram of a system that illustrates probetransmission, according to an embodiment of the disclosure.

FIG. 4 is a data structure diagram that illustrates probe records, andmanners of identifying coded probes from the probe records, according toan embodiment of the disclosure.

FIG. 5 is a graph of a system that illustrates identifying andcorrecting loop errors, according to an embodiment of the disclosure.

FIG. 6 is a block diagram of a model of a control system to correct alocal clock frequency of a machine, according to an embodiment of thedisclosure.

FIG. 7 is a block diagram of an adaptive filter bank, according to anembodiment of the disclosure.

FIG. 8 is a flowchart that illustrates an exemplary process forimplementing an embodiment of the disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

In order to ensure clocks are synchronized to a high degree of accuracy,at least two parameters are controlled on a continuous (e.g., periodic,but ongoing) basis. First, offset, which describes a difference by whichtwo clocks differ in their estimation of time, is determined andadjusted for. For example, if clock A estimates that it is currently4:05 pm, and clock B estimates that it is currently 4:15 pm, then anoffset of 10 minutes exists between clock A and clock B. Second,frequency drift (also referred to as “drift”), which describes adifference in the frequency of two clocks, is determined and adjustedfor. For example, inexpensive clocks have quartz components and quartzis sensitive to temperature and vibration. As temperature changes, orvibration occurs, the frequency of a clock using quartz will change overtime, and this change is tracked as described herein.

The systems and methods disclosed herein may be used to improve accuracyof clock synchronization to a degree of nanoseconds due to finetunedsystems and processes for estimating offset and drift of any givenclock. Further, clocks are guaranteed to not deviate from a timeindicated by a reference clock beyond an upper and lower bound, theupper and lower bound being within twenty-five standard deviations ofsynchronization error, which is on the order of one microsecond. Offsetand drift estimations contain noise, which is introduced based on, e.g.,queuing delays of packets used to estimate offset and drift, as well asthe effect of network operation (e.g., latency introduced duringtransmission). The finetuned systems and processes of estimating offsetand drift as described herein are resistant to noise from offset anddrift estimates (e.g., using advanced filtering techniques), and thusenable highly precise clock synchronization. These systems and methodsachieve such accuracy even when the response of each clock to controlinput may be different (e.g., responses differ between differentclocks), unknown (e.g., a clock's response is not known a priori), andtime-varying (e.g., a clock's response changes over time). This allowsthese systems and methods to be applied to commodity, off-the-shelf, andinexpensive clocks. Further, these systems and methods can beimplemented without a requirement for implementing any extra hardware.

Figure (FIG. 1 is a graph of a network including a coordinator forexecuting an embodiment of the disclosure. Network 100 includes machines110, which are the nodes of the graph. The term “machine” refers to anydevice that maintains a clock or produces timestamps, such as a physicalor virtual machine, a server, a server blade, a virtual machine, and thelike. Each of machines 110 includes a local clock (e.g., as implementedin a computer processing unit (CPU) of a machine, or as implemented in adevice that is operably coupled to a machine, such as a networkinterface card (NIC) of a machine). As depicted, network 100 is a meshnetwork, where each machine 110 is linked to each other machine 110 byway of one or more links (some links omitted for clarity). However,network 100 may be any other type of network. For example, network 100may be a network where machines are serially connected on a wire, or maybe in any other configuration. The network may be a large networkspanning multiple physical regions (e.g., New York to San Francisco), ora small network, such as a network within a single server blade. In anembodiment, network 100 may be a network of clocks on one or moreprinted circuit boards.

The communication links between any pair of machines are represented asan edge 120 between the nodes in the graph. Each edge 120 typicallyrepresents multiple paths between any two machines 110. For example, thenetwork 110 may include many additional nodes other than the machines110 that are shown, so that there may be multiple different pathsthrough different nodes between any pair of machines 100.

Network 100 additionally includes coordinator 130 and reference clock140. In this example, coordinator 130 commands machines 110 to obtainnetwork observations by probing other machines 110, as will be describedin greater detail below with respect to FIG. 3. Coordinator 130 maystore, or cause to be stored, records of those network observations, aswill be described in greater detail below with respect to FIG. 4.Coordinator 130 may additionally transmit control signals to machines110. The term control signal, as used herein, may refer to a signalindicating that the frequency of a local clock of a machine is to beadjusted by a specified amount (thus correcting a drift of the localclock), and may also refer to a signal indicating that a time indicatedby a local clock a machine is to be adjusted by a specified amount (thuscorrecting an offset of the local clock).

In an embodiment, coordinator 130 stores, either within a machinehousing coordinator 130 or within one or more machines of network 100, agraph that maps the topology of network 100. The graph may include adata structure that maps connections between machines of network 100.For example, the graph may map both direct connections between machines(e.g., machines that are next hops from one another, either physicallyor logically), as well as indirect connections between machines (e.g.,each multi-hop path that can be taken for a communication, such as aprobe, to traverse from one machine to another). The graph mayadditionally include network observations corresponding to each edge inthe graph (e.g., indicating probe transit times for probes that crossedthe edge, and/or additional information, such as information depicted inFIG. 4).

One of the machines contains a reference clock 140. Reference clock 140is a clock to which the clocks within the machines of network 100 are tobe synchronized. In an embodiment, reference clock 140 is a highlycalibrated clock that is not subject to drift, which is contained in amachine 110 that is different than the other machines to besynchronized. In another embodiment, reference clock 140 may be anoff-the-shelf local clock already existing in a machine 110 that willact as a master reference for the other machines 110, irrespective ofwhether reference clock 140 is a highly tuned clock that is accurate to“absolute time” as may be determined by an atomic clock or some otherhighly precise source clock. In such scenarios, coordinator 130 mayselect which machine 110 will act as the master reference arbitrarily,or may assign the reference machine based on input from anadministrator. The reference clock may be a time source, such as aglobal positioning system (GPS) clock, a precision time protocol (PTP)Grandmaster clock, an atomic clock, or the like, in embodiments wherethe reference clock 140 is accurate to “absolute time.” As will bedescribed in greater detail with respect to FIGS. 6-8, coordinator 130may use reference clock 140 when calibrating a control signal. Bysignaling corrections to frequency and/or offset based on referenceclock 140, coordinator 130 achieves high-precision synchronization ofthe local clocks of machines 110 to the reference clock 140.

While only one reference clock 140 is depicted in FIG. 1, in anembodiment, multiple reference clocks 140 may be present. For example,additional reference clocks may be used for redundancy in case offailures of the reference clocks or the machines housing them. Asanother example, machines 110 may be divided into multiple groups (e.g.,based on settings applied by an administrator of network 100). Forexample, network 100 may be configured to divide machines 110 intogroups that each have a predefined number, or percentage, of machines100, which may improve performance or implementation. Within each group,one machine may be nominated to be a reference, and the clock of thatmachine will thus be used as the reference clock for the remainingmachines. Further, the groups may nominate one group as a referencegroup, thus leading to the reference group's reference clock acting as areference for all groups. The reference clock and group may be nominatedautomatically consistent with clock nomination described herein, or maybe input by a network administrator. As an example, where a networkincludes five thousand machines, and an administrator programs thenetwork to divide the machines into groups that each hold one hundredmachines (e.g., based on specifying one hundred machines, or byspecifying a percentage), there will be fifty groups of machines, eachincluding one hundred machines. One of the one hundred machines will bea reference machine, and the other ninety-nine of the one hundredmachines will be sync to the reference machine's clock. Moreover, of thefifty groups, one will be a reference group, and the other fourty-ninegroups will sync to the reference group.

Coordinator 130 may be implemented in a stand-alone server, may beimplemented within one or more of machines 110, or may have itsfunctionality distributed across two or more machines 130 and/or astandalone server. Coordinator 130 may be accessible by way of a link120 in network 100, or by way of a link to a machine or server housingcoordinator 130 outside of network 100. Reference clock 140 may beimplemented within coordinator 130, or may be implemented as a separateentity into any of machines 110, a standalone server within network 100,or a server or machine outside of network 100.

FIG. 2 is a data flow diagram for correcting clock frequency and/oroffset, according to an embodiment of the disclosure. The left column ofFIG. 2 describes activities of a coordinator (e.g., coordinator 130) inachieving highly precise clock synchronization by correcting clockfrequency (i.e., drift) and/or offset; the right column describesactivities of machines (e.g., machines 110). FIG. 2 can be thought of asincluding three phases—a first phase where network observations are madeby having machines probe other machines of a network (e.g., network100), a second phase where the network observations are used to estimateoffset and drift of the machines, and a third phase where frequencyand/or offset is compensated/corrected in order to achieve highlyprecise clock synchronization between the machines.

As part of the first phase, data flow 200 begins with a coordinator(e.g., coordinator 130) assigning 202 machine pairs. The term pair, asused herein, refers to machines that send probes to one another for thepurpose of collecting network observations. As used herein, the termnetwork observations may refer to observable qualities of a network(e.g., effect of network operation, as defined below; queuing delays;observable drift; offset; etc.). The term probes, as used herein, refersto an electronic communication transmitted from one machine to anothermachine, where the electronic communication is timestamped at its timeof transmission from a sending machine, and at its time of receipt at areceiving machine. The timestamps may be applied by any component of themachines that are configured to apply timestamps, such as respectiveCPUs of the sending and receiving machines and/or respective NICs thatare a part of, or that are operably coupled to, the sending andreceiving machines. As will be described in further detail with respectto FIG. 3, a single machine typically is paired with multiple othermachines. When assigning machine pairs, the coordinator may assign amachine to pair with a number of machines, the number being less thanall machines in the network. In an embodiment, the number and pairingsof machines may be predefined or may dynamically change based on networkconditions (e.g. congestion, latency, etc.). The machines may beselected at random, or through a deterministic algorithm.

Data flow 200 progresses by coordinator 130 instructing the pairedmachines to transmit 204 probes to one another, which will also bedescribed in further detail with respect to FIG. 3. The networkobservations collected from the probe transmissions are collected 206into probe records. The term probe record, as used herein, may refer toa data structure including network observations obtained from the probetransmissions, such as the identity of a transmitting machine and areceiving machine, a transmit timestamp, a receive timestamp, etc. Thetransit time for a probe may be determined based on the transmittimestamp and the receive timestamp. Probe records are described infurther detail below with respect to FIG. 4. While the embodimentdescribed here and depicted in FIG. 3 indicates that the coordinatorcollects the probe records, in an embodiment, some or all of themachines may each collect probe records pertaining to probes transmittedto or from them, and may themselves perform processing on the proberecords.

After the probe records are collected, the coordinator (e.g.,coordinator 130) enters the second phase of using the collected proberecords to estimate offset and/or drift for the machines (e.g., machines110). In this example, to achieve accurate estimations, the coordinatorfirst filters 208 the probe records to identify coded probes. The termcoded probes, as used herein, refers to probes that correspond to proberecords that are not affected by noise, such as delay caused fromqueuing the probes. One manner in which the coordinator identifies codedprobes is described in further detail with respect to FIG. 4. The subsetof probe records that correspond to coded probes may be referred to ascoded probe records. In an embodiment where probe records are collectedat a given machine, that given machine may perform the filtering 208 ofthe probe records collected by that given machine.

Data flow 200 continues by applying 210 a classifier to the coded proberecords. The classifier may be a machine learning model trained throughsupervised learning. An example classifier is a support vector machine(“SVM”). The coordinator may input upper and lower bound points derivedfrom coded probe data (i.e., samples of transit time) from two pairedmachines over a time period. The output of the classifier is a linearfit to the transit time data with a slope and intercept. Data flow 200then continues with the coordinator estimating 212 the drift betweenpairs of machines. In an embodiment, the coordinator estimates drift tobe equivalent to, or a function of, the slope of the linear fit (i.e.,estimate of rate of change of transit time). The coordinator may alsoestimate offset using the intercept of the linear fit.Determining/estimating offset may be performed in a similar manner todoing so for drift wherever disclosed. In an embodiment where proberecords are collected at a given machine, that given machine may performthe applying 210 of the classifier to the probe records collected bythat given machine, and the estimating 212 of the drift between thepairs of machines.

The drift estimate may not be completely accurate because, while thecoded probes did not suffer from queuing delay, the coded probes mayhave suffered from the effect of network operation. The effect ofnetwork operation, as used herein, may refer to noise caused bycomponents of a network. For example, a link or gateway between twopaired machines may introduce latency or jitter that affects the driftestimation. In an embodiment, the coordinator uses 214 the networkeffect based on frequency drift estimations across three or moremachines. Further details for using 214 the network effect will bedescribed with respect to FIG. 5 below.

The coordinator sends 216 observations to a control loop of a localclock of a machine, e.g., by applying a filter to the estimated driftthat is based on the effect of the network operation, or by feeding theestimated drift and the effect of the network operation to a machinelearning model, the output of which is the absolute drift. Here,“absolute” drift or offset are relative to the reference clock. Furtherdetails about the control loop and how the coordinator estimates theabsolute drift are described in further detail below with respect toFIGS. 6-8. After estimating the absolute drift, the coordinator maydetermine whether to correct 218 the clock frequency in real-time ornear real-time. Absolute offsets may also be corrected 218, or thecoordinator may perform 220 an offline correction. How to determinewhether to correct in real-time, or offline, is described further belowwith reference to FIGS. 6-7.

In addition to correcting clock frequency and/or offset, process 200recurs periodically for each machine pair to ensure that any new offsetand drift that has occurred after correcting clock frequency and/oroffset is continuously corrected. For example, process 200 may occurperiodically (e.g., every two seconds) to ensure synchronization acrossthe network (e.g., network 100) is maintained.

FIG. 3 is a block diagram of a system that illustrates probetransmission, according to an embodiment of the disclosure. System 304depicts a network (e.g., network 100) with machines 310. Machines 310have the same functionality described with respect to machines 110 ofFIG. 1. System 304 depicts the transmitting 204 of probes between pairedmachines described with respect to FIG. 2. The number of machines thatare paired to a given machine may be a number that is fixed by anadministrator. In an embodiment, coordinator 130 may dynamically adjustthe number of machines that are paired to each given machine for probingbased on network conditions. For example, if network connectivity isabove a threshold quality (e.g., based on a low amount of latency),coordinator 130 may reduce a number of machines that are paired forprobing proportionally to the quality of network connectivity. On theother hand, as network connectivity quality deteriorates, coordinator130 may increase the number of machines that are paired for probing toeach given machine. The number of machines that are paired may also varybased on network size.

When selecting which machines should be paired to a given machine,coordinator 130 may randomly determine each machine to which the givenmachine should be paired. In an embodiment, coordinator 130 non-randomlydetermines pairings based on ease of computation, accuracy (e.g., clocksynchronization accuracy as dictated by the network graph), and loadbalancing across each machine 110. Coordinator 130 may judiciouslydetermine pairings based on design choice, with an administratorselecting pairings, or selecting parameters that cause certain pairingsto be selected. Further, coordinator 130 may instruct a larger number ofpairings to occur for machines that have a high level of diversity,relative to a number of pairings for machines that have a low level ofdiversity. As used herein, the term “diversity” may refer to a largenumber of paths from which a probe may cross within network 100 to reacha machine from another machine; the higher the number of paths, thehigher the diversity.

While FIG. 3 does not depict a number of probes that pass between eachpair of machines, it is noted that probes may occur at a particularfrequency or period of time, which may vary based on one or moreparameters. For example, coordinator 130 may command a machine that hasa high diversity to transmit a large number of probes to one or morepaired machines, each down a different path, as compared to a machinethat has a low diversity, which may transmit a lower number of probes.Coordinator 130 thus may command machines to transmit machines to pairedmachines at a frequency that varies proportionate to diversity.

As depicted, the coordinator (e.g., coordinator 130) determines 202 thatmachine 310C of machines 310 is paired with machine 310B, machine 310F,machine 310H, and machine 310I, as shown by the dashed lines. Thus,machine 310C transmits 204 probes to machines 310B, 310F, 310H, and310I, and receives probes from those same machines. The term “exchange”is used herein to describe scenarios where paired machines transmit andreceive probes from one another. As used herein, the term exchange doesnot imply a timing aspect, such as a requirement that machines aretransmitted simultaneously or responsive to one another.

In an embodiment, network 100 may be a trustless system, such as asystem facilitating a blockchain network. In such an embodiment, some ofmachines 110 may misbehave and misrepresent data used to determineoffset and/or drift. In such a scenario, in addition to the probesdiscussed above and with respect to FIG. 4, coordinator 130 may instructmachines 110 to probe, at low frequency (e.g., once per every ten, orone hundred, or one thousand probes sent in normal course), a largernumber of machines (e.g., one-third of the machines of network 100).Coordinator 130 may determine therefrom whether the data obtained fromthe larger number of machines is consistent with the smaller number ofmachines that are probed in normal course (e.g., within a thresholdtolerance amount of drift and/or offset), and may alert an administratoror take corrective action if inconsistent probe data is detected.

FIG. 4 is a data structure diagram that illustrates probe records, andmanners of identifying coded probes from the probe records, according toan embodiment of the disclosure. Data structure 400 includes exemplaryprobe records for a plurality of probes. While data structure 400 isdepicted to include probe records for a single transmitting machine “A”(which is, e.g., a machine 110 of network 100) and a single receivingmachine “B,” any number of transmitting or receiving machines may haveprobe records stored within data structure 400. Column 410 includesidentifiers for probes, each identifier 1 through 6 corresponding to aprobe record for a probe. Column 420 indicates which machine transmitteda probe indicated by a given probe record. Column 420, as depicted,indicates that a transmitting machine labeled “A” transmitted eachprobe; however, this is merely exemplary and various transmittingmachines may be identified in column 420.

Column 430 indicates which machine received a probe indicated by a givenprobe record. Column 430, as depicted, indicates that a receivingmachine labeled “B” received each probe; however, this is merelyexemplary and various receiving machines may be identified in column430. Column 440 indicates a transmit time of a probe. The transmit timeis a time that is timestamped either by the transmitting machine itself(e.g., a CPU of transmitting machine “A” of network 100), or by aninterface or device operably coupled to the transmitting machine (e.g.,a NIC of transmitting machine “A” of network 100). Similarly, column 450indicates a receive time of a probe, which is a timestamp by thereceiving machine or, e.g., a NIC of the receiving machine. In anembodiment, a machine having a single CPU may have a plurality of NICs.In such an embodiment, coordinator 130 may cause the multiple NICs of amachine (e.g., the receiving machine) to sync to a clock of the CPU ofthe machine (e.g., by having the CPU synchronize its time to the time ofthe NIC, using the NIC as a reference machine as described herein), andthen have the other NICs synchronize to the CPU, thus causing themultiple NICs of the machine to be synchronized.

The coordinator may command machines to transmit probes with a specifiedor predetermined time interval between probes. As used herein, the term“transmission time spacing” (6) refers to the specified interval orpredetermined time interval between the transmission times of twoprobes. The interval may be a constant value or may be dynamicallyselected by the coordinator based on network conditions (e.g., if thenetwork is congested, a longer transmission time spacing may beselected). As can be seen in FIG. 4, probe 1 is sent at time T1, whichmay be an arbitrary time or a time specified by the coordinator. Probe 2is sent at time T1+δ, as the coordinator has instructed transmittingmachine A to transmit a probe to receiving machine B at one or more timeintervals. Further probes may be commanded by the coordinator to betransmitted from transmitting machine A to receiving machine B from thereference point of time T1; however, for ease of illustration, only twoare shown in FIG. 4. Similarly, probes 3 and 4 are sent at times T2 andT2+δ, respectively, and probes 5 and 6 are sent at times T3 and T3+δ,respectively.

Probe IDs 1 and 2, 3 and 4, and 5 and 6 are paired to illustrate how thecoordinator determines whether a pair of probes are coded probes. Codedprobes are probes that are transmitted with a specific spacing of δ, orwithin a threshold distance from δ. That is, the probes are coded basedon the space between each probe. Delay in timestamping probes may becaused by queues at a transmitting machine 420 and/or at a receivingmachine 430 or through intermediate nodes. Coded probes are thus pairsof probes that are consecutively transmitted by a same transmittingmachine 420, and received by a same receiving machine 430, with receivetimes that differ by δ, or within a threshold margin of δ (toaccommodate minimal differences in delay between the two probes). Thatis, the transit times of two coded probes is approximately the same.While pairs are primarily used to describe coded probes, this is merelyexemplary; coded probes may be triplets, quadruplets, etc., of probeswith a spacing of δ.

Probes 1 and 2 show a scenario where two probes do not form coded probesbecause probe 1 has a transit time of TT, but probe 2 has a transit timeof TT+D (D representing a delay), where D is greater than a thresholdmargin. That is, probe 2 has a transit time that is D longer than probe2. Probes 3 and 4 show a scenario where two probes do not form codedprobes because probe 3 has a transit time that is D longer than probe 4.Probes 5 and 6, however, are coded probes because they have the sametransit times (to within an acceptable threshold).

In an embodiment, data structure 400 is stored in memory directlyaccessible to coordinator 130 (e.g., local memory of a machine runningcoordinator 130). In another embodiment, data structure 400 isdistributed across machines 110, where each machine stores a local datastructure 400 for probes exchanged between that machine and othermachines. Various processing is described below with respect to FIGS.5-8 that uses information of data structure 400; this processing may beperformed by coordinator 130, but may also, or alternatively, beperformed by machines 110. Where machines 110 are performing processing(e.g., identifying coded probes, applying a support vector machine,etc.), if one machine is overburdened with other processing, anothermachine in the pair may retrieve data of data structure 400 of theoverburdened machine, and perform the processing on that overburdenedmachine's behalf.

As was described above with respect to FIG. 2, coded probe records maybe input into a classifier, such as an SVM classifier, from which driftmay be estimated. However, a drift estimate may nonetheless beinaccurate, but correctable by using the network effect. FIG. 5 is agraph of a system that illustrates identifying and correcting looperrors, according to an embodiment of the disclosure. Machines 510include machine 1, machine 2, and machine 3, which together form anetwork loop. Machines 510 have the same functionality described abovewith respect to machines 110 and 310. While only three machines aredepicted, this is merely for convenience; any number of machines mayform a network loop. Links 520 connect the machines of the network loop,where links 520-1 connect machine 1 to machine 2, links 520-2 connectmachine 2 to machine 3, and links 520-3 connect machine 3 to machine 1.Each link may represent multiple different paths between each pair ofmachines.

The numbers over each link 520 are the drift between the two machinesthat are connected by each respective link in arbitrary units. Thus,link 520-1 reflects a drift of +20 units for the drift of machine 1relative to the drift of machine 2, link 520-2 has a drift of −15 unitsbetween machines 2 and 3, and link 520-3 reflects a drift of +5 unitsbetween machines 3 and 1. The sum of the drifts around a given loop(referred to as the loop drift error, which is a result of networkeffect applied to frequency) is reflective of error in an estimatedclock drift. Thus, if there was no loop drift error, then the sum of thedrifts of all links in the loop would be 0 units. However, as depicted,the sum of the drifts is 10 units (in that 20−15+5=10), which may becaused by inaccurate clock estimates, which can be corrected using thenetwork effect. The coordinator may assign a given machine to be part ofmultiple loops when assigning pairs. The coordinator may combine allloops for different pairs of machines to estimate clock drift moreaccurately using the network effect. When assigning pairs, thecoordinator is not constrained by a need for path symmetry; the timetaken (or number of hops) to go from machine 1 to machine 2 need not bethe same as the time taken to go from machine 2 to machine 1. In anembodiment, some of the loops includes reference clock 140, thusensuring the network effect is determined with respect to the referenceclock. In an embodiment (e.g., where coordinator 130 is not present),the network effect can be used without reference to a reference clock,where each clock determines its frequency drift, and a statisticaloperation (e.g., average) is taken to determine the loop drift error.These loop drift errors around different loops are used to adjust theabsolute drift of the machines in the loops. For example, the loop drifterror for a loop may be allocated among the different machines in theloop.

FIG. 6 is a block diagram of a model of an adaptive stochastic controlsystem to correct a local clock frequency of a machine, according to anembodiment of the disclosure. Control loop 600 is used to adjust thefrequency of a local clock 615 of a machine 610 by way of a controlsignal 604. The machine 610 may be a machine of network 100, andincludes the same functionality described above with respect to machines110, machines 310, and machines 510. Coordinator module 630 is depictedas part of machine 610, but may alternatively sit wholly or partially ina separate coordinator (e.g., coordinator 130), as described above withrespect to FIG. 1.

Coordinator module 630 estimates the absolute offset and absolute drift602 of machine 610, as described above with respect to FIGS. 2-5. Theseare absolute quantities because they are measured against the referenceclock that is connected to a source of absolute time. The control loop600 also includes a filter 660 and a controller 670. Filter 660 may be apredefined filter (e.g., a Kalman filter), a filter selected from anadaptive filter bank based on observations, a machine learning model,etc. Kalman filters and adaptive filter banks are discussed in furtherdetail with respect to FIG. 7; use of a machine learning model isdiscussed in further detail with respect to FIG. 8.

The purpose of filter 660 is two-fold: first, to reduce noise in thedrift and offset estimations and, second, to extrapolate the naturalprogression of the clock. Process 200 (from FIG. 2) repeats on aperiodic basis (e.g., every two seconds), and thus control loop 600loops periodically as well. In an embodiment, clock offsets areestimated in the middle of the period (e.g., 1 second into a 2-secondperiod), whereas control signals happen at the end of the period (e.g.,at the 2-second mark of the 2-second period). Thus, filter 660, inaddition to reducing noise in the estimate, extrapolates to output 603filtered offset and drift values that are accurate at the time ofcontrol. Filtered offset and drift are received by controller 670.Controller 670 outputs 604 a frequency (and offset) adjustment signal tolocal clock 615 of machine 610, the adjustment being reflective offrequency and offset value changes in local clock 615 to remove offsetand drift from local clock 615. The frequency and offset adjustments arealso fed back to filter 660 as parameters for the filter, in addition tothe estimated offset and drift for the filter, on a subsequent cycle ofthe control loop. In this control loop, the plant under control isdetermined by the state variables {absolute offset, absolute drift} ofthe local machine and an adaptive stochastic controller is used tocontrol the plant. As will be described with respect to FIG. 7 below,adaptive stochastic control refers to adjusting control signals based ona likelihood that a given adjustment is a correct adjustment, ascompared to other possible adjustments; as control signals are applied,actual adjustments are observed, and probabilities that each possiblecontrol signal will lead to a correct adjustment are adjusted.

FIG. 7 is a block diagram of an adaptive filter bank, according to anembodiment of the disclosure. The term adaptive filter bank, as usedherein, may refer to a collection of candidate filters, each of which isbest suited to remove noise from signals based on the type and degree ofnoise. For example, some noise can be observed, such as the networkobservations discussed with respect to FIGS. 1-6 (e.g., queuing delays,effect of network operation, loop errors, etc.). Some noise, however, isinherent in the state of the machines, and is unknown to control loop600 (e.g., noise variations in response to control input acrossdifferent makes and models of equipment). Noise that is unknown isreferred to herein as state noise.

Filter 760, which includes the functionality of filter 660 as describedabove with respect to FIG. 6, includes a bank of candidate filters 761(also referred to herein as an adaptive filter bank), which may beKalman filters. Each of candidate filters 761 corresponds to a differentlevel of state noise. Filter selection module 762 is a stochasticselection module, in that it selects a filter from candidate filter 761by calculating a probability for each candidate filter being a best fit,and by then selecting the candidate filter with the best fit. Initially,filter selection module 762 receives observed noise, and uses theobserved noise to select a highest probability candidate filter 761,which is used to filter the estimated drift and offset 702, and outputthe filtered drift and offset 703 to the controller 670. Using adaptivestochastic control, it is possible that initially filter selectionmodule 762 may find that all filters are equally likely, and may selecta filter arbitrarily. After selecting a filter and observing how localclock 615 reacts to a control signal, filter selection module 762adjusts the likelihood that each candidate filter 761 best applies.Thus, as the control signal and further information about the networkobservations are fed into filter 760 over time, the selection of anappropriate candidate filter 761 eventually converges to a best matchingcandidate filter.

As was discussed with reference to FIG. 2, when deriving the controlsignal to be transmitted to the local clock of a machine, the correctionmay be performed in real-time, thus resulting in a real-time controlsignal (or near real-time control signal). In an embodiment, correctionsmay be performed offline, such as where observation noise is muchsmaller than state noise. For example, the coordinator may determinewhether observation noise is a predefined amount or factor smaller thanthe state noise. In response to determining that the observation noiseis a predefined amount or factor smaller than the state noise, thecoordinator may perform the adjustment offline (or in batched fashion);otherwise, the coordinator may perform the adjustment in real-time ornear-real-time and thus cause a control signal to quickly be sent to thelocal clock. An administrator of the system may set parameters thatdetermine when offline corrections will be made, and may elect thatoffline corrections are not used at all.

FIG. 8 is a flowchart that illustrates an exemplary process forimplementing an embodiment of the disclosure. Process 800 begins with acoordinator (e.g., coordinator 130) accessing 802 probe records (e.g.,probe records of data structure 400) for probes transmitted at differenttimes between pairs of machines in a network, such as a mesh network.The probe records (e.g., as indexed by probe ID 410) each identify themachine that transmitted the probe (e.g., transmitting machine 420), themachine that received the probe (e.g., receiving machine 430), andprovide information sufficient for determining a transit time of theprobe based on a transmit timestamp from the transmitting machine and areceive timestamp from the receiving machine. At least one of themachines exchanging probes produces timestamps based on the referenceclock. The probe records may be stored within network 100, within amachine executing coordinator 130, or distributed across severalmachines.

For different pairs of machines, the coordinator (or the machinesthemselves) estimates 804 the drift between the pair of machines basedon the transit times of probes transmitted between the pair of machines.For example, coordinator 130 derives coded probe records from the proberecords and applies an SVM to the coded probe records to obtain a linearfunction, the slope of which is used to estimate the drift between thepair of machines.

In an embodiment, coordinator 130 optionally estimates 806 an absolutedrift of each machine based on the estimated drifts between pairs ofmachines. In an embodiment, to determine absolute drift of a givenmachine, when the coordinator assigns pairs, each machine is paired withthe reference machine. In an alternative embodiment, when thecoordinator assigns pairs, each machine is paired with at least onemachine that is paired with the reference machine. In a furtheralternative embodiment, each machine is at least indirectly paired withthe reference machine, such that a chain of paired machines, asindicated by the network graph, eventually pairs a paired machine withthe reference machine. As described above, reference clock 140 may beintegrated into one or more of machines 110, such that one or moremachines 110 have a clock that is used as a reference clock.

Coordinator 130 may additionally estimate an absolute offset of eachmachine. For example, transit times of coded probes feed into theaforementioned linear function, and thus, based on their transit times,the intercept of the linear function is determined. In an embodiment,the absolute offset may be determined to an accuracy on the order ofnanoseconds.

As process 800 continues, for different loops of at least threemachines, the coordinator calculates 808 a loop drift error based on asum of the estimated drifts between pairs of machines around one or moreloops (e.g., loop 500), as discussed above with reference to FIG. 5. Thecoordinator then adjusts 810 the estimated absolute drifts of themachines based on the loop drift errors (e.g., using control loop 600,as discussed above with reference to FIGS. 6-7), and transmitsinstructions to at least one of the machines (e.g., machine 610) toadjust a frequency of that machine's local clock (e.g., local clock 615)based on the estimated absolute drift for that machine.

As was described above with reference to FIG. 6-7, adjusting thefrequency of a machine's local clock may include the coordinatorapplying adaptive stochastic control (e.g., by way of stochasticdetermination of a filter of filters 761). As used herein, the termstochastic control may refer to applying control signals based on aprobability associated with candidate control signals (as dictated bycandidate filters of an adaptive filter bank). As a result, state of theplant that is controlled may be modelled by the absolute offset andabsolute drift of that machine, as state of the local clock feeds in tocontrol loop 600. In another embodiment, the coordinator may apply amachine learning model as part of the controller to adjust the frequencyof the machine's local clock, where the machine learning model receivesthe estimated absolute drift as an input, and outputs a control signal(e.g., control signal 604) to adjust the frequency of that machine'slocal clock (e.g., local clock 615).

The foregoing description of the embodiments of the disclosure may beimplemented in a software module installed for use with off-the-shelfclocks, including inexpensive and inaccurate clocks, such as quartzclocks, for bringing such clocks into highly precise synchronization.The foregoing description of embodiments of the disclosure have beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed, butmerely illustrates different examples. It should be appreciated that thescope of the disclosure includes other embodiments not discussed indetail above. Persons skilled in the relevant art can appreciate thatmany modifications and variations are possible in light of the abovedisclosure, without departing from the spirit and scope as defined inthe appended claims. Therefore, the scope of the disclosure should bedetermined by the appended claims and their legal equivalents.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsdescribed herein is intended to be illustrative, but not limiting, ofthe scope of the disclosure, which is set forth in the following claims.

What is claimed is:
 1. A computer program product for syntonizingmachines in a network, the computer program product comprising anon-transitory machine-readable medium storing computer program code forperforming a method, the method comprising: accessing probe records forprobes transmitted at different times between pairs of machines in thenetwork; each probe record identifying the machine that transmitted theprobe, identifying the machine that received the probe, and sufficientto determine a transit time of the probe based on a transmit timestampfrom the transmitting machine and a receive timestamp from the receivingmachine; and one of the machines produces timestamps based on areference clock; for different pairs of machines, estimating a driftbetween the pair of machines based on the transit times of probestransmitted between the pair of machines; for different loops of atleast three machines in the network, calculating a loop drift errorbased on a sum of the estimated drifts between pairs of machines aroundthe loop; and adjusting estimated absolute drifts of the machines basedon the loop drift errors, wherein the absolute drift of a machine isdefined relative to the reference clock.
 2. The computer program productof claim 1 wherein the probe records comprise, for pairs of machines,probe records for probes transmitted in both directions between themachines in each of the pairs.
 3. The computer program product of claim1, wherein estimating the drift between the pair of machines is based onthe transit times of coded probes, wherein coded probes are pairs ofprobes transmitted from one of the machines in the pair to the other ofthe machines in the pair, the transmit timestamps of the pair of probesare within a predetermined time interval of each other, and the transittimes of the pair of probes are also within a predetermined deviation ofeach other.
 4. The computer program product of claim 3, whereinestimating the drift between the pair of machines comprises applying asupport vector machine classifier to probe records for the coded probesfor the pair of machines.
 5. The computer program product of claim 1,wherein which machines transmit probes between each other changes basedon network conditions.
 6. The computer program product of claim 1,wherein each machine in the network transmits probes to a predefinedmaximum number of other machines.
 7. The computer program product ofclaim 1, further comprising: determining and instructing the machines asto which machines transmit probes to which other machines.
 8. Thecomputer program product of claim 1, wherein the method furthercomprises estimating an absolute offset for each machine based on thetransit times of probes transmitted between pairs of machines.
 9. Thecomputer program product of claim 1 wherein the absolute offsets areestimated to an accuracy of nanoseconds or better.
 10. The computerprogram product of claim 1, wherein the method further comprises:causing at least one of the machines to adjust a frequency of thatmachine's local clock based on the estimated absolute drift for thatmachine.
 11. The computer program product of claim 10, wherein adjustingthe frequency of that machine's local clock comprises applying adaptivestochastic control to adjust the frequency of that machine's localclock.
 12. The computer program product of claim 10, wherein adjustingthe frequency of that machine's local clock comprises applying adaptivestochastic control to the local clock, wherein a state of the localclock is determined by an absolute offset and the absolute drift of thatmachine.
 13. The computer program product of claim 10, wherein adjustingthe frequency of that machine's local clock comprises applying adaptivestochastic control to adjust the frequency of that machine's local clockand the adaptive stochastic control includes a Kalman filter.
 14. Thecomputer program product of claim 10, wherein adjusting the frequency ofthat machine's local clock comprises applying adaptive stochasticcontrol to adjust the frequency of that machine's local clock and theadaptive stochastic control includes an adaptive filter bank.
 15. Thecomputer program product of claim 10, wherein adjusting the frequency ofthat machine's local clock comprises applying a machine learning modelto adjust the frequency of that machine's local clock, wherein themachine learning model receives the estimated absolute drift as an inputand outputs a control signal to adjust the frequency of that machine'slocal clock.
 16. The computer program product of claim 10, whereinadjusting the frequency of that machine's local clock syntonizes thatmachine's local clock to the reference machine regardless of noiseintroduced in network observations.
 17. The computer program product ofclaim 16, wherein the noise introduced in network observations isattributable to instability in that machine's local clock from at leastone of temperature variations and vibration variations.
 18. The computerprogram product of claim 1, wherein at least one of the transmittimestamps and the receive timestamps includes at least one of a networkinterface card (NIC) timestamp, a computer processor unit (CPU)timestamp, a virtual machine (VM) timestamp, a container timestamp, andan antenna timestamp.
 19. The computer program product of claim 1,wherein the absolute drifts are estimated to an accuracy of tennanoseconds per second or better.
 20. The computer program product ofclaim 1 wherein the network is a mesh network.
 21. The computer programproduct of claim 1, wherein the machines in the network form a group ofmachines, wherein the network comprises a plurality of groups ofmachines including the group of machines, and wherein input from anadministrator defines the groups of machines.
 22. The computer programproduct of claim 21, wherein the administrator defines at least one ofthe reference clock, and a reference group of machines.